Skip to content

Implemented Inclusion of min/max for Floats, Doubles and Items Sorted Views. - #547

Merged
leerho merged 2 commits into
masterfrom
fix_min_max
Apr 17, 2024
Merged

leerho merged 2 commits into
masterfrom
fix_min_max

Conversation

@leerho

@leerho leerho commented Apr 16, 2024

Copy link
Copy Markdown
Member

Also fixed a problem I discovered in the deserialization for Boolean quantiles, but it would also have shown up with other types. The data was ok, but the max value would have been incorrect.

Also fixed some problems with some of the tests.

Along the way I also replaced "back-slash-n" with platform independent "LS" in the files I touched.

Also moved some functions like getCDF(..) and getPMF(...) into parent interfaces as default functions. Now the three parent interfaces for Doubles SV, Floats SV and Items SV are all code-parallel.

Also fixed a problem I discovered in the deserialization for Boolean
quantiles, but it would also have shown up with other types. The data
was ok, but the max value would have been incorrect.

Also fixed some problems with some of the tests.

Along the way I also replaced "back-slash-n" with platform independent
"LS" in the sketches I touched.

Also moved some functions like getCDF(..) and getPMF(...) into parent
interfaces as default functions. Now the three parent interfaces for
Doubles SV, Floats SV and Items SV are all code-parallel.
@leerho
leerho requested a review from jmalkin April 16, 2024 05:40
@leerho

leerho commented Apr 16, 2024

Copy link
Copy Markdown
Member Author

Changes by file:

/kll/

KllDirectCompactItemsSketch

  • Fix for getMaxItem

KllDoublesSketch

  • Propagation of new SV constructor signature

KllFloatsSketch

  • Propagation of new SV constructor signature

KllHeapDoublesSketch

  • Removed //TODO

KllItemsSketch

  • Fixed a problem in toString(...)

KllMemoryValidate

  • add code doc for better understanding

/quantiles/

DoublesSketch

  • Propagation of new SV constructor signature

ItemsUtil

  • Removed obsolete code comment

/quantilescommon/

DoublesSketchSortedView

  • Revised the main constructor to include Min/Max fix.

  • Added separate test constructor also with Min/Max fix

  • Removed unneeded class members for min/max

FloatsSketchSortedView

  • Revised the main constructor to include Min/Max fix.

  • Added separate test constructor also with Min/Max fix

  • Removed unneeded class members for min/max

GenericSortedView

  • Moved getCDF() and getPMF() from ItemsSketchSortedView to here as an interface default implementation. This is consitent with how floats and dobles are implemented.

IncludeMinMax

  • A new class that consolidates all new include Min and Max code for floats, doubles and item in one place.

ItemsSketchSortedView

  • Revised the main constructor to include Min/Max fix.

  • Added separate test constructor also with Min/Max fix

  • Removed unneeded class members for min/max

/req/

ReqSketch

  • Propagation of new SV constructor signature

/test/...

/kll/

KllDirectCompactItemsSketch

  • Minor restructuring and added diagnostic print statements

KllMiscDoublesTest

  • Redesigned a test to accommodate new min/max adjustments

  • Apply "/n" to "LS" corrections

KllMiscFloatsTest

  • Redesigned a test to accommodate new min/max adjustments

  • Apply "/n" to "LS" corrections

KllMiscItemsTest

  • Redesigned a test to accommodate new min/max adjustments

  • Apply "/n" to "LS" corrections

/quantilescommon/

IncludeMinMaxTest

  • tests for the new IncludeMinMax class

@leerho leerho changed the title Implemented fix for min/max for Floats, Doubles and Items Sorted Views. Implemented Inclusion of min/max for Floats, Doubles and Items Sorted Views. Apr 16, 2024
@leerho

leerho commented Apr 16, 2024

Copy link
Copy Markdown
Member Author

We changed the behavior of the sorted view to ensure the min and max values are always represented, which, we believe, will provide a more practical information to users of these distributions without distorting the resulting data. -- @jmalkin

@jmalkin jmalkin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few minor things to clean up and extra asserts in new tests. All really small stuff.

Overall, I think this is a very useful change.

final int offset = srcLevels[0];
System.arraycopy(srcQuantiles, offset, quantiles, 0, numItems);
//Remove free space from both itemsArray and levels array
final int[] myLevelsArr = new int[srcLevelsArr.length];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unclear why using this instead of srcNumLevels+1 as before, but gonna trust it's fine either way

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without studying closely, the srcNumLevels + 1 makes it seem like we are increasing the size of the levelsArr, when we are not. It is the same size as the original, but with different offsets.

this.totalN = sk.getN();
}

//Used for testing

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Constructors like this are ... not great. We should really be trying to find ways to test w/o special constructors like this. Not gonna "require changes" because of it, but just noting it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of the tests would be very difficult to reproduce without this.

* @throws IllegalArgumentException if sketch is empty.
*/
double[] getPMF(T[] splitPoints, QuantileSearchCriteria searchCrit);
//double[] getPMF(T[] splitPoints, QuantileSearchCriteria searchCrit);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove commented-out line?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, will fix.

GenericSortedView.validateItems(splitPoints, getComparator());
final double[] buckets = getCDF(splitPoints, searchCrit);
final int len = buckets.length;
for (int i = len; i-- > 1; ) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can just use int i = buckets.length here. clearly not wrong, but not necessary to define len :)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, will fix.

System.arraycopy(quantilesIn, 0, adjQuantiles, offset, lenIn);
System.arraycopy(cumWeightsIn,0, adjCumWeights, offset, lenIn);

//Adjust the low end if required.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

asked on slack and we don't need to adjust the next item's weight cuz it's already cumulative here. but can we add a short comment about that for future reference?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep. done.

* @author Lee Rhodes
*/
@SuppressWarnings("unused")
//@SuppressWarnings("unused")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove commented out line if not needed?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep. Done.

}
final int topIn = quantiles.length - 1;
final int topAdj = adjQuantiles.length - 1;
assertEquals(adjQuantiles[topAdj], maxItem);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think it'd be good to assert that the adjQuantiles array is 2 longer than the original quantiles array in here, too.

The printfs might be convient when debugging but they're basically no-ops in tests (and I treat them as such when reviewing). We want to make sure we assert on the expected
end-user behavior. And in this case, that's that the arrays are longer because we add in min and max.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@leerho
leerho merged commit ce7f478 into master Apr 17, 2024
@leerho
leerho deleted the fix_min_max branch April 17, 2024 01:59
stojkomilos added a commit to stojkomilos/datasketches-cpp that referenced this pull request Nov 7, 2026
…etches

kll_sketch and quantiles_sketch keep the exact min and max, but
get_quantile(0) and get_quantile(1) read only the retained items, which
compaction can drop the min and max from. Since 6.0.0, datasketches-java
adds them back in its sorted views (IncludeMinMax, apache/datasketches-java#547).

Port that step: quantiles_sorted_view::include_min_max() adds a missing
min or max as an entry of weight 1, taken from its neighbor, so the total
weight does not change. kll_sketch and quantiles_sketch call it from
get_sorted_view(). REQ does not change, because its get_rank() does not
read the sorted view.

The min and max tests build 1000 sketches and check the quantiles at
ranks 0 and 1, also on a deserialized sketch and on its merge with a
second sketch.

Co-authored-by: Isaac <no-reply@databricks.com>
stojkomilos added a commit to stojkomilos/datasketches-cpp that referenced this pull request Nov 10, 2026
…EQ sketches

The KLL, quantiles and REQ sketches keep the exact min and max, but
get_quantile(0) and get_quantile(1) read only the retained items, and
compaction can drop the min and max from them. Since 6.0.0,
datasketches-java adds them back in its sorted views (IncludeMinMax,
apache/datasketches-java#547).

Port that step: the sketches call the new
quantiles_sorted_view::convert_to_cummulative(min_item, max_item) from
get_sorted_view(). It appends a missing max with weight 1 taken from the
entry before it, and adds a missing min with weight 1 taken from the
entry after it, so the total weight does not change. The min is added in
the same pass that makes the weights cumulative, so the entries shift
without a second pass.

REQ get_rank() summed the weights in the compactors and did not read the
sorted view. Move it to the cached sorted view, as in KLL, quantiles and
Java ReqSketch, and remove req_compactor::compute_weight(), which has no
other caller.

For non-arithmetic types, the view and get_quantile() refer to the items
of the sketch, now also to its min and max. Document that they must not
be used after the sketch is updated, merged, assigned to, moved or
destroyed.

Keep get_sorted_view() as fast as before: the view reserves room for the
min and max, but only for a non-empty sketch. add() returns early for an
empty range (a KLL sketch usually has one or two empty levels once n is
much larger than k) and uses emplace_back() instead of push_back() of a
temporary.

The new tests check the quantiles at ranks 0 and 1 in estimation mode,
for floats and strings, and after deserialize and merge, plus the view
with the min, the max, both or neither missing.

Co-authored-by: Isaac <no-reply@databricks.com>
stojkomilos added a commit to stojkomilos/datasketches-cpp that referenced this pull request Nov 10, 2026
…EQ sketches

The KLL, quantiles and REQ sketches keep the exact min and max, but
get_quantile(0) and get_quantile(1) read only the retained items, and
compaction can drop the min and max from them. Since 6.0.0,
datasketches-java adds them back in its sorted views (IncludeMinMax,
apache/datasketches-java#547).

Port that step: the sketches call the new
quantiles_sorted_view::convert_to_cummulative(min_item, max_item) from
get_sorted_view(). It appends a missing max with weight 1 taken from the
entry before it, and adds a missing min with weight 1 taken from the
entry after it, so the total weight does not change. The min is added in
the same pass that makes the weights cumulative, so the entries shift
without a second pass.

REQ get_rank() summed the weights in the compactors and did not read the
sorted view. Move it to the cached sorted view, as in KLL, quantiles and
Java ReqSketch, and remove req_compactor::compute_weight(), which has no
other caller.

For non-arithmetic types, the view and get_quantile() refer to the items
of the sketch, now also to its min and max. Document that they must not
be used after the sketch is updated, merged, assigned to, moved or
destroyed.

The view reserves room for the min and max, so adding them does not
reallocate it. It reserves only for a non-empty sketch, so an empty
sketch does not allocate.

The new tests check the quantiles at ranks 0 and 1 in estimation mode,
for floats and strings, and after deserialize and merge, plus the view
with the min, the max, both or neither missing.

Co-authored-by: Isaac <no-reply@databricks.com>
stojkomilos added a commit to stojkomilos/datasketches-cpp that referenced this pull request Nov 10, 2026
…EQ sketches

The KLL, quantiles and REQ sketches keep the exact min and max, but
get_quantile(0) and get_quantile(1) read only the retained items, and
compaction can drop the min and max from them. Since 6.0.0,
datasketches-java adds them back in its sorted views (IncludeMinMax,
apache/datasketches-java#547).

Port that step: the sketches call the new
quantiles_sorted_view::convert_to_cummulative(min_item, max_item) from
get_sorted_view(). It appends a missing max with weight 1 taken from the
entry before it, and adds a missing min with weight 1 taken from the
entry after it, so the total weight does not change. The min is added in
the same pass that makes the weights cumulative, so the entries shift
without a second pass.

REQ get_rank() summed the weights in the compactors and did not read the
sorted view. Move it to the cached sorted view, as in KLL, quantiles and
Java ReqSketch, and remove req_compactor::compute_weight(), which has no
other caller.

For non-arithmetic types, the view and get_quantile() refer to the items
of the sketch, now also to its min and max. Document that they must not
be used after the sketch is updated, merged, assigned to, moved or
destroyed.

The view reserves room for the min and max, so adding them does not
reallocate it. It reserves only for a non-empty sketch, so an empty
sketch does not allocate.

The new tests check the quantiles at ranks 0 and 1 in estimation mode,
for floats and strings, and after deserialize and merge, plus the view
with the min, the max, both or neither missing.

Co-authored-by: Isaac <no-reply@databricks.com>
stojkomilos added a commit to stojkomilos/datasketches-cpp that referenced this pull request Nov 10, 2026
…EQ sketches

The KLL, quantiles and REQ sketches keep the exact min and max, but
get_quantile(0) and get_quantile(1) read only the retained items, and
compaction can drop the min and max from them. Since 6.0.0,
datasketches-java adds them back in its sorted views (IncludeMinMax,
apache/datasketches-java#547).

Port that step: the sketches call the new
quantiles_sorted_view::convert_to_cummulative(min_item, max_item) from
get_sorted_view(). It appends a missing max with weight 1 taken from the
entry before it, and adds a missing min with weight 1 taken from the
entry after it, so the total weight does not change. The min is added in
the same pass that makes the weights cumulative, so the entries shift
without a second pass.

REQ get_rank() summed the weights in the compactors and did not read the
sorted view. Move it to the cached sorted view, as in KLL, quantiles and
Java ReqSketch, and remove req_compactor::compute_weight(), which has no
other caller.

For non-arithmetic types, the view and get_quantile() refer to the items
of the sketch, now also to its min and max. Document that they must not
be used after the sketch is updated, merged, assigned to, moved or
destroyed.

The view reserves room for the min and max, so adding them does not
reallocate it. It reserves only for a non-empty sketch, so an empty
sketch does not allocate.

The new tests check the quantiles at ranks 0 and 1 in estimation mode,
for floats and strings, and after deserialize and merge, plus the view
with the min, the max, both or neither missing.

Co-authored-by: Isaac <no-reply@databricks.com>
stojkomilos added a commit to stojkomilos/datasketches-cpp that referenced this pull request Nov 10, 2026
…EQ sketches

The KLL, quantiles and REQ sketches keep the exact min and max, but
get_quantile(0) and get_quantile(1) read only the retained items, and
compaction can drop the min and max from them. Since 6.0.0,
datasketches-java adds them back in its sorted views (IncludeMinMax,
apache/datasketches-java#547).

Port that step. A missing min or max is added to the sorted view as an
entry of weight 1, taken from its neighbor, so the total weight does not
change. The levels are sorted, so get_sorted_view() finds out if the min
is retained by comparing it with the first item of each level. If not,
it adds the min with add_min() before the levels, so no entry has to
move, and add() does not merge the first level into the lone min.
convert_to_cummulative(max_item) takes the weight of the min from the
next entry and appends a missing max. The view reserves room for both,
but only for a non-empty sketch.

REQ get_rank() summed the weights in the compactors and did not read the
sorted view. Move it to the cached sorted view, as in KLL, quantiles and
Java ReqSketch, and remove req_compactor::compute_weight(), which has no
other caller.

For non-arithmetic types, the view and get_quantile() refer to the items
of the sketch, now also to its min and max. Document that they must not
be used after the sketch is updated, merged, assigned to, moved or
destroyed.

The new tests check the quantiles at ranks 0 and 1 in estimation mode,
for floats and strings, and after deserialize and merge, plus the view
with the min, the max, both or neither missing, and with empty ranges.

Co-authored-by: Isaac <no-reply@databricks.com>
stojkomilos added a commit to stojkomilos/datasketches-cpp that referenced this pull request Nov 11, 2026
…EQ sketches

The KLL, quantiles and REQ sketches keep the exact min and max, but
get_quantile(0) and get_quantile(1) read only the retained items, and
compaction can drop the min and max from them. Since 6.0.0,
datasketches-java adds them back in its sorted views (IncludeMinMax,
apache/datasketches-java#547).

Port that step. A missing min or max is added to the sorted view as an
entry of weight 1, taken from its neighbor, so the total weight does not
change. The levels are sorted, so get_sorted_view() finds out if the min
is retained by comparing it with the first item of each level. If not,
it adds the min with add_min() before the levels, so no entry has to
move, and add() does not merge the first level into the lone min.
convert_to_cummulative(max_item) takes the weight of the min from the
next entry and appends a missing max. The view reserves room for both,
but only for a non-empty sketch.

REQ get_rank() summed the weights in the compactors and did not read the
sorted view. Move it to the cached sorted view, as in KLL, quantiles and
Java ReqSketch, and remove req_compactor::compute_weight(), which has no
other caller.

For non-arithmetic types, the view and get_quantile() refer to the items
of the sketch, now also to its min and max. Document that they must not
be used after the sketch is updated, merged, assigned to, moved or
destroyed.

The new tests check the quantiles at ranks 0 and 1 in estimation mode,
for floats and strings, and after deserialize and merge, plus the view
with the min, the max, both or neither missing, and with empty ranges.

Co-authored-by: Isaac <no-reply@databricks.com>
stojkomilos added a commit to stojkomilos/datasketches-cpp that referenced this pull request Nov 13, 2026
kll_sketch keeps the exact min and max, but get_quantile(0) and
get_quantile(1) read only the retained items, and compaction can drop
the min and max from them. Since 6.0.0, datasketches-java adds them back
in its sorted views (IncludeMinMax, apache/datasketches-java#547).

Port that step for KLL. A missing min or max is added to the sorted view
as an entry of weight 1, taken from its neighbor, so the total weight
does not change. The levels are sorted, so get_sorted_view() finds out
if the min is retained by comparing it with the first item of each
level. If not, it adds the min with add_min() before the levels, so no
entry has to move, and add() does not merge the first level into the
lone min. convert_to_cummulative(max_item) takes the weight of the min
from the next entry and appends a missing max. The view reserves room
for both, but only for a non-empty sketch. The quantiles and REQ
sketches do not change.

For non-arithmetic types, the view and get_quantile() refer to the items
of the sketch, now also to its min and max. Document that they must not
be used after the sketch is updated, merged, assigned to, moved or
destroyed.

The new tests check the KLL quantiles at ranks 0 and 1 in estimation
mode, for floats and strings, and after deserialize and merge, plus the
view with the min, the max, both or neither missing, and with empty
ranges.

Co-authored-by: Isaac <no-reply@databricks.com>
stojkomilos added a commit to stojkomilos/datasketches-cpp that referenced this pull request Nov 13, 2026
kll_sketch keeps the exact min and max, but get_quantile(0) and
get_quantile(1) read only the retained items, and compaction can drop
the min and max from them. Since 6.0.0, datasketches-java adds them back
in its sorted views (IncludeMinMax, apache/datasketches-java#547).

Port that step for KLL. A missing min or max is added to the sorted view
as an entry of weight 1, taken from its neighbor, so the total weight
does not change. The levels are sorted, so get_sorted_view() finds out
if the min is retained by comparing it with the first item of each
level. If not, it adds the min with add_min() before the levels, so no
entry has to move, and add() does not merge the first level into the
lone min. convert_to_cummulative(max_item) takes the weight of the min
from the next entry and appends a missing max. The view reserves room
for both, but only for a non-empty sketch. The quantiles and REQ
sketches do not change.

For non-arithmetic types, the view and get_quantile() refer to the items
of the sketch, now also to its min and max. Document that they must not
be used after the sketch is updated, merged, assigned to, moved or
destroyed.

The new tests check the KLL quantiles at ranks 0 and 1 in estimation
mode, for floats and strings, and after deserialize and merge, plus the
view with the min, the max, both or neither missing, and with empty
ranges.

Co-authored-by: Isaac <no-reply@databricks.com>
stojkomilos added a commit to stojkomilos/datasketches-cpp that referenced this pull request Nov 13, 2026
kll_sketch keeps the exact min and max, but get_quantile(0) and
get_quantile(1) read only the retained items, and compaction can drop
the min and max from them. Since 6.0.0, datasketches-java adds them back
in its sorted views (IncludeMinMax, apache/datasketches-java#547).

Port that step for KLL. A missing min or max is added to the sorted view
as an entry of weight 1, taken from its neighbor, so the total weight
does not change. The levels are sorted, so get_sorted_view() finds out
if the min is retained by comparing it with the first item of each
level. If not, it adds the min with add_min() before the levels, so no
entry has to move, and add() does not merge the first level into the
lone min. convert_to_cummulative(max_item) takes the weight of the min
from the next entry and appends a missing max. The view reserves room
for both, but only for a non-empty sketch. The quantiles and REQ
sketches do not change.

For non-arithmetic types, the view and get_quantile() refer to the items
of the sketch, now also to its min and max. Document that they must not
be used after the sketch is updated, merged, assigned to, moved or
destroyed.

The new tests check the KLL quantiles at ranks 0 and 1 in estimation
mode, for floats and strings, and after deserialize and merge, plus the
view with the min, the max, both or neither missing, and with empty
ranges.

Co-authored-by: Isaac <no-reply@databricks.com>
stojkomilos added a commit to stojkomilos/datasketches-cpp that referenced this pull request Nov 13, 2026
kll_sketch keeps the exact min and max, but get_quantile(0) and
get_quantile(1) read only the retained items, and compaction can drop
the min and max from them. Since 6.0.0, datasketches-java adds them back
in its sorted views (IncludeMinMax, apache/datasketches-java#547).

Port that step for KLL. A missing min or max is added to the sorted view
as an entry of weight 1, taken from its neighbor, so the total weight
does not change. The levels are sorted, so get_sorted_view() finds out
if the min is retained by comparing it with the first item of each
level. If not, it adds the min with add_min() before the levels, so no
entry has to move, and add() does not merge the first level into the
lone min. convert_to_cummulative(max_item) takes the weight of the min
from the next entry and appends a missing max. The view reserves room
for both, but only for a non-empty sketch. The quantiles and REQ
sketches do not change.

For non-arithmetic types, the view and get_quantile() refer to the items
of the sketch, now also to its min and max. Document that they must not
be used after the sketch is updated, merged, assigned to, moved or
destroyed.

The new tests check the KLL quantiles at ranks 0 and 1 in estimation
mode, for floats and strings, and after deserialize and merge, plus the
view with the min, the max, both or neither missing, and with empty
ranges.

Co-authored-by: Isaac <no-reply@databricks.com>
stojkomilos added a commit to stojkomilos/datasketches-cpp that referenced this pull request Nov 13, 2026
kll_sketch keeps the exact min and max, but get_quantile(0) and
get_quantile(1) read only the retained items, and compaction can drop
the min and max from them. Since 6.0.0, datasketches-java adds them back
in its sorted views (IncludeMinMax, apache/datasketches-java#547).

Port that step for KLL. A missing min or max is added to the sorted view
as an entry of weight 1, taken from its neighbor, so the total weight
does not change. The levels are sorted, so get_sorted_view() finds out
if the min is retained by comparing it with the first item of each
level. If not, it adds the min with add_min() before the levels, so no
entry has to move, and add() does not merge the first level into the
lone min. convert_to_cummulative(max_item) takes the weight of the min
from the next entry and appends a missing max. The view reserves room
for both, but only for a non-empty sketch. The quantiles and REQ
sketches do not change.

For non-arithmetic types, the view and get_quantile() refer to the items
of the sketch, now also to its min and max. Document that they must not
be used after the sketch is updated, merged, assigned to, moved or
destroyed.

The new tests check the KLL quantiles at ranks 0 and 1 in estimation
mode, for floats and strings, and after deserialize and merge, plus the
view with the min, the max, both or neither missing, and with empty
ranges.

Co-authored-by: Isaac <no-reply@databricks.com>
stojkomilos added a commit to stojkomilos/datasketches-cpp that referenced this pull request Nov 16, 2026
kll_sketch keeps the exact min and max, but get_quantile(0) and
get_quantile(1) read only the retained items, and compaction can drop
the min and max from them. Since 6.0.0, datasketches-java adds them back
in its sorted views (IncludeMinMax, apache/datasketches-java#547).

Port that step for KLL. A missing min or max is added to the sorted view
as an entry of weight 1, taken from its neighbor, so the total weight
does not change. The levels are sorted, so get_sorted_view() finds out
if the min is retained by comparing it with the first item of each
level. If not, it adds the min with add_min() before the levels, so no
entry has to move, and add() does not merge the first level into the
lone min. convert_to_cummulative(max_item) takes the weight of the min
from the next entry and appends a missing max. The view reserves room
for both, but only for a non-empty sketch. The quantiles and REQ
sketches do not change.

For non-arithmetic types, the view and get_quantile() refer to the items
of the sketch, now also to its min and max. Document that they must not
be used after the sketch is updated, merged, assigned to, moved or
destroyed.

The new tests check the KLL quantiles at ranks 0 and 1 in estimation
mode, for floats and strings, and after deserialize and merge, plus the
view with the min, the max, both or neither missing, and with empty
ranges.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants